Multiverse Analyses

2025-06-17

Multiverse Analyses

by Giulia Calignano
University of Padova
Department of Developmental and Social Psychology, Psicostat

MB Logo Psicostat Logo Psicostat Logo

Outline

  1. Part 1: Understanding the Multiverse

  2. Part 2: MB2-P Pupillometry Case Study

  3. Co-authors:
    Melanie S. Schreiner • Alvin Tan • Marlena Mayer • Robert Hepach and the MB2-P Team —

Part 1: Understanding the Multiverse

What is Multiverse Analysis?

  • “A systematic exploration of all reasonable analytical paths” (Steegen et al., 2016)
  • Enhances transparency and reduces researcher degrees of freedom
  • A garden of analytical forking paths: grounded in theory

Why It Matters for Developmental Science

  • Universal Applicability: works across contexts and methods
  • Flexible Implementation: supports exploratory and confirmatory designs
  • Credibility Enhancement: addresses replication challenges
  • ManyBabies Alignment: fits with goals of transparency and rigor

The Multiverse Landscape

  • Data Inclusion/Exclusion: who/what to include?
  • Preprocessing Methods: cleaning, transforming
  • Variable Operationalization: defining constructs
  • Statistical Models: which methods and why

Visualizing the Multiverse

Specification Curve

Volcano Plot

Volcano Plot

Part 2: MB2-P Pupillometry Case Study

Study Design Overview

  • Cross-lab pupillometry with over 30 labs worldwide
  • Age cohorts: toddlers and adults
  • Condition × Outcome × Time modeling
  • From: Calignano, Girardi, Altoè., 2024 First steps into the pupillometry multiverse of developmental science. Behav Res Methods.

ManyBabies 2: Theory of Mind in Infancy

MB2 – Do toddlers and adults engage in spontaneous Theory of Mind (ToM)?

MB2 – ToM: Paradigm

MB2 – ToM

MB2 – ToM

MB2P – Secondary Data Analysis

MB2P – Secondary Data Analysis

MB2P – Secondary Data Analysis

  • MB2P investigates responses to goal-congruent vs. goal-incongruent outcomes
  • Is the child surprised if the protagonist (bear) acts incongruently with the goal (following the mouse)?
  • Measures: pupil dilation and looking time during events

The MB2P Preregistration and the Multiverse Approach

The MB2P Preregistration and Data Simulation

  • Data Collection by MB2 completed in 2023. Raw data will be shared after Stage 1.
  • Analyses will focus on the second test trial.
  • Data Simulation mirrors expected data structure to pre-register processing and analysis strategies.
  • Simulated dataset includes:
    • Participant ID, Two age cohorts , Timestamps or durations, x/y coordinates, Pupil size (left/right), Lab ID, Conditions and outcomes

MB2-P Multiverse Structure

  • Five Forking Dimensions:
    • Pupil plausibility
    • Gaze within screen
    • Filtering via moving average
    • Baseline correction window (1s, 0.5s, 0.25s)
    • Participant exclusion (none, trial 2, trial 1&2)

Simulated Data

Density Plot by Condition × Outcome × Age

DF1: Pupil Plausibility

DF1: Pupil Plausibility

DF2: Gaze Filtering

DF2: Gaze Filtering

DF3: Moving Average Filter

DF3: Moving Average Filtering

DF4: Baseline Correction

`geom_smooth()` using formula = 'y ~ x'

DF4: Baseline correction

DF5: Participant Inclusion Criteria

# A tibble: 2 × 2
  df5          n
  <chr>    <int>
1 excluded 47200
2 included   800

Specification Curve of the Multiverse’ Models

Specification Curve of Model Estimates

Volcano plot of the Multiverse’ Models

Volcano Plot of Effect Estimates

Conclusions

  • The Explorative Multiverse analysis approaches shows how results shift across processing paths: their fragility/robustness

  • The Multiverse reveal consistent vs. fragile effects

  • Aligned with suggestions from Calignano et al. (2024) and Steegen et al. (2016)

Next steps

  • Implementation Generalized Additive Mixed effect Models (Gamm) non linear relationship between pupil variation and time
  • Shiny app to allow the whole community to explore and touch data
  • Towards a consensus for a transparent and reproducible data management and preprocessing pipelines, thanks to collaborative multilab efforts

References 1/3

  • Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016). Increasing transparency through a multiverse analysis. Perspectives on Psychological Science

  • Dragicevic, P., Jansen, Y., Sarma, A., Kay, M., & Chevalier, F. (2019, May). Increasing the transparency of research papers with explorable multiverse analyses. In proceedings of the 2019 chi conference on human factors in computing systems

References 2/3

  • Simonsohn, U., Simmons, J. P., & Nelson, L. D. (2020). Specification curve analysis. Nature Human Behaviour

  • Calignano, G., Girardi, P., & Altoè, G. (2024). First steps into the pupillometry multiverse of developmental science. Behavior Research Methods

  • Sirois, S., Brisson, J., Blaser, E., Calignano, G., Donenfeld, J., Hepach, R., … & Valenza, E. (2023). The pupil collaboration: A multi-lab, multi-method analysis of goal attribution in infants. Infant Behavior and Development

References extra

  • STAY TUNED: Altoè, G., Gambarota, F., Girardi, P., Vesely, A., Calignano, G., Pastore, M., & Finos, L. (in progress). Multiverse analysis in R: Exploratory and inferential approaches.

Thank You

  • Questions? Feedback?

MB Logo Psicostat Logo Psicostat Logo